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REHABILITATION WITH MUSIC 
RELATED APPLICATIONS 

The present application claims the benefit under 35 USC 119(e) of US provisional 
application 60/542,022 filed on February 5, 2004, the disclosure of which is incorporated 
5 herein by reference. 

FIELD OF THE INVENTION 

The present invention relates to rehabilitation, training and/or exercise using music. 

BACKGROUND OF THE INVENTION 
Rehabilitation of an injured person and/or training in the use of specific muscles 
10 requires tedious repetition of muscle movements. Generally such training is guided by a 
physical therapist or personal trainer teaching the person what moves to make and making sure 
that the person performs the moves. 

People tend to get bored and tired of performing the exercises, especially if the motion 
is complicated, strenuous or causes pain. 
15 Generally listening to music is considered enjoyable. Music typically transfers a rhythm 

to Hie listener and gives the user an urge to move. It is common practice to combine motion and 
music, for example as in dancing. Additionally, exercise and music are known to be combined, 
for example as in aerobics. 

Music has also been used as background for physical therapy and rehabilitation. 
20 SUMMARY OF THE INVENTION 

An aspect of some embodiments of the invention, relates to devices that combines 
rehabilitation with music. In an exemplary embodiment of the invention, the device tracks a 
patient's motion and gives feedback to the patient (and/or a trainer ot other person) using 
changes in music, for example indicating mistakes or generating comment such as an indication 
25 of speed of motion or smoothness of motion. In an exemplary embodiment of the invention, the 
device generates music to guide the person in performing rehabilitation. In one example, the 
device translates complex trajectories in to music and music parameters that can be readily 
perceived. In another example, music is used to assist in synchronizing multiple body parts. In 
another example, music is used to give advance warning or a reminder of a change in motion, 
30 for example, a raising tempo indicating an upcoming change in direction of motion. In an 
exemplary embodiment of the invention, music is used as a guiding language for assisting in 
communication between an operator and a patient. 3h one example, an operator's motions are 
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recorded spatially and using music and feedback to the operator on the quality of performance 
of such motions by a patient are provided using music. 

In an exemplary embodiment of the invention, music is used for patients with cognitive 
impairment of the verbal system, which may not be able to understand, follow and/or generate 
5 verbal instructions or comment. 

In some embodiments of the invention, music is used for physiological purposes, for 
example to motivate or calm down a patient, for example automatically in response to 
physiological sensing thereof. Optionally, music is used to convey general instructions, rather 
than instructions relating to a particular movement In one example, music is used to indicate 
..10 (e.g., a chime) that a series of exercises is reaching a last set 

In some embodiments of the invention music is used as an integral part of rehabilitation 
to add interest to the rehabilitation and/or fill in for times when a patient is too tired to perform 
an activity or during short rest periods. Optionally, music is used to provide motivation to a 
patient, e.g., a feeling that the exercises are actually having an effect on the real world. 
15 In an exemplary embodiment of the invention, the use of music is used to convey a 

sense of time and/or timing to a rehabilitating person. 

In some embodiments of the invention, a patient is required to move in a specific path 
or a desired location. Optionally alternatively or additionally, the patient is required to move at 
a specific rate or follow a particular tempo and/or rhythm, 
20 hi some embodiments of the invention, various sensors, such as position sensors, 

accelerometers or cameras are used to track the motion of a patient (or other user). 

ha some embodiments of the invention, a part of the device is attached to the patient and 
moves with the patient Optionally, the device guides motion by prohibiting or hindering 
certain motion, for example exerting force to cause the patient to move in a manner which 
25 follows a specific path and/or use the same tempo as the pre-selected music. 

In an exemplary embodiment of the invention, a pre-selected music is played for the 
user responsive to the user motion. Optionally the music is played correctly if the user motion 
is performed correctly, for example correct timing and/or in the correct position. In some 
embodiments of the invention, the music is distorted, for example played ofibeat or off tune, if 
30 the user motion is performed incorrectly. In some embodiments of the invention, additional 
sounds are added to signify errors in the motion, for example beeps or a verbal response. 
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In some embodiments of the invention, the device coordinates between motions of 
different parts of a patient's body, for example different channels indicating motions of 
different body parts. 

In some embodiments of the invention, the device coordinates between motions of 
5 multiple users. In one example, the multiple users are multiple patients, for example, organized 
as a band, with each patient's rehabilitation activities sounding a different instrument. 
Differences in patient's abilities are optionally bridged by suitable devices, for example, one 
patient maybe required to keep to a tempo within 10% and other be required to keep a motion 
to be circle-like, without restriction on speed or rhythm and with a precision of 50%, Li both 
10 cases, if the requirements are met the devices will produce synchronized correct music. A 
potential advantage of such a setup is that a patient can feel that he is a productive part of a 
group. This may also provide motivation. In another example,- the multiple users are a patient 
and an instructor. In another example, the multiple users are a patient and a layperson, such as a 
family member, which may assist in ensuring family support. 
15 In an exemplary embodiment of the invention, games for rehabilitation are provided. In 

one example, the rehabilitation device is played as an instrument, for example, a patient plays 
drums by moving the device. Optionally, virtual reality techniques are used to provide 
feedback. In one example, a screen (or helmet) shows how the patient is hitting the dram. 
Optionally, the generated music is used a$ in karaoke where the patient needs to add his 
20 instrument to a pre-recorded band. 

In some embodiments of the invention, another visual feedback is used, for example 
showing a person doing the correct motions or giving encouragement using a visual device, for 
example a computer screen or a television. 

There is thus provided in accordance with an exemplary embodiment of the invention, 
25 apparatus, comprising: 

at least one sensor which generates an indication of a morion of a user; 
a rhythmic audio generator; and 

a controller configured to control said generator to modify an audio output thereof 
according to a program of rehabilitation of a human. Optionally, said controller modifies said 
30 audio according to said indication. 

BRIEF DESCRIPTION OF FIGURES 
Particular exemplary embodiments of the invention will be described with reference to 
the following description of embodiments in conjunction with the figures, wherein identical 
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structures, elements or parts which appear in more than one figure are generally labeled with a 
same or similar number in all the figures in which they appear, wherein: 

Fig. 1 A is a schematic illustration of a rehabilitation system 100 usable for integrating 
music and rehabilitation, in accordance with an exemplary embodiment of the invention. 
5 Fig. IB is a block diagram of a portion of a controller of Fig. 1 A, in accordance with an 

exemplary embodiment of the invention; 

Fig. 2 is a schematic illustration of a configuration including cameras and/or sensors for 
tracking user motion according to an exemplary embodiment of the invention; 

Fig. 3 is a schematic illustration of a supportive input device according to an exemplary 
10 embodiment of the invention; 

Fig. 4 is a flow diagram of a method of rehabilitating a user in accordance to an 
exemplary embodiment of the invention; and 

Fig. 5 is a flow diagram of an alternative method of rehabilitating a user in accordance 
to an exemplary embodiment of the invention. 
15 DETAILED DESCRIPTION OF EMBODIMENTS 

Exemplary Device design 

Fig. 1 A is a schematic illustration of a rehabilitation system 100 usable for integrating 
music and rehabilitation, in accordance with an exemplary embodiment of the invention. 

In an exemplary embodiment of the invention, rehabilitation system 100 comprises a 
20 controller, such as a personal computer 170 and an input device 160 for tracking user motion 
and inputting it into computer 170. 

In an exemplary embodiment of the invention, input device 160 is optionally a standard 
input device of a computer, for example a joystick (not shown), a mouse 140, a keyboard 130, a 
trackball (not shown), a drawing tablet (not shown), a touch screen 180, a pedal 150 or any 
25 other standard input device. Alternatively or additionally, input device 160 is optionally a 
specially designed device, for example an articulated robot arm or other input devices as 
described in tLS. provisional patent application 60/542,022. In some embodiments of the 
invention, input device 1 60 is grasped by the user. Alternatively, input device 160 is attached to 
the user or supports a limb of the user depending on the type of motion performed with the 
30 input device as described below. Optionally, different input devices with different degrees of 
freedom are used depending on the muscles of the user that need to be trained and the level of 
control the user has of the muscles. 
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In some embodiments of the invention, multiple input devices 160 are provided, for 
example, for providing input form two or more limbs. 

In an exemplary embodiment of the invention, computer 170 comprises an output 
device, for example display 120, which may be used for giving instructions and/or responses to 
5 the user. 

Tn an exemplary embodiment of the invention, rehabilitation system 100 comprises 
speakers 190 used for playing music and/or other audio signals. Other audio sources may be 
used as well, for example a head set, for example a Bluetooth wireless bead set. 

Fig. IB is a block diagram of a portion of controller 170, in accordance with an 
1 0 exemplary embodiment of the invention. This portion may be implemented, for example, using 
software, using hardware and/or using a combination thereof. In some embodiments, other 
structures than the one shown in Fig, IB are used, for example, an electronic circuit or a neural 
network 

A sensor analysis module 172 receives signals from sensors, for example position 
15 sensors and pressure sensors. Data from encoders of input device 160 may be converted, for 
example, into position information. 

A motion analysis module 174 analyses motion as reflected by the positions of body 
limbs, for example, determining smoothness of motion or spatial shape of the motion, 

A processor 176 then applies one of the methods described belo^v, for example, for 
20 controlling an audio generator 1 77* Optionally, a database 178 of desired motions and/or music 
is used for applying such methods. 

Fig. 2 is a schematic illustration of a rehabilitation configuration 200, which can serve 
as an input device 160 for rehabilitation system 100, according to an exemplary embodiment of 
the invention. In an exemplaiy embodiment of the invention, a disabled user 290, for example 
25 with a physical handicap, is seated in a chair 210. Optionally, user 290 is supported by a body 
rest 220 if user 290 needs support In an exemplary embodiment of the invention, chair 210 is 
fixated with fixation bars 230 to a control device 240. Control device 240 is used by user 290 
to train a damaged limb, for example a hand or a foot In an exemplary embodiment of the 
invention, control device 240 is connected to computer 170 and transmits details of the motion 
30 performed by user 290. In some embodiments of the invention, control device 240 is controlled 
by computer 170 and trains user 290 by guiding the limb of user 290 to perform a specific 
motion. In some embodiments of the invention, the user's limb is attached to control device 
240, for example using a strap, to prevent control device 240 from slipping away from the 
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grasp of user 290. Control device 240 can be of various types, for example, it can be a ball with 
motors and/or a brake and an optional extending handle. Encoders may be provided on the ball 
and/or the handle, or on motors thereof* In another embodiment, control device 240 is an 
articulated robotic arm. Force sensors may be provided in the robotic arm. Pressure sensors 
5 may also be attached to parts of the body in contact with parts of system 1 00 or in contact with 
other items. 

In some embodiments of the invention, rehabilitation device 200 comprises cameras or 
sensors 270 to detect the motion of user 290. Optionally, user 290 wears markers 260, for 
example fiduciary markers or transmitters to allow recording of the motion of a whole limb or 

10 other parts of the body of user 290. hi some embodiments of the invention, the markers are 
passive markers that allow an external device to record motion based on them. Alternatively or 
additionally, the markers are active devices that sense and transmit a characteristic of the user 
motion, for example an accelerator meter that transmits acceleration, speed or an inclination 
sensor that senses and transmits the inclination of the sensor on the user relative to the 

15 perpendicular, hi an exemplary embodiment of the invention, cameras or sensors 270 transmit 
details of the motion of user 290 to computer 170 in order to Tecord the motion and prepare 
feedback based on the motion. In some embodiments of the invention, cameras or sensors 270 
track an absolute position of user 290 as a function of time. Alternatively ot additionally, 
cameras or sensors 270 record a user's speed or acceleration. 

20 Depending on the type of sensor used, one or more of the following informational items 

about the motion maybe determined: speed, position, smoothness, acceleration, applied force 
and/or characteristic frequencies, for example of tremor. These characteristics can be, for 
example of particular parts of a body (e.g., a hand) or of multiple body parts taking part in a 
same motion (e.g„ hand, wrist elbow and shoulder). 

25 La some embodiments of the invention, sensors 270 comprise one or more physiological 

sensors, which record physiological parameters of a user, for example pulse rata EMG, EEG, 
muscle tension and/or breathing rate. In an exemplary embodiment of the invention, 
rehabilitation system 100 responds immediately to these parameters, for example stop the 
exercise if the user is over stressed. Alternatively, the parameters may serve for general user 

30 progress reports. In an exemplary embodiment of the invention, the pulse Tate is used as a 
feedback to vary music tempo, for example, to match to a speed, for providing feedback to the 
patient and/or to provide feedback to a caretaker. The actual signal may be indicative of limits 
rather than absolute values. In an exemplary embodiment of the invention, if stressing of the 
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patient is determined, calming music is played, to reduce such tension. In other embodiments, 
for example, sensing EEG and EMG, the sensed signals relate directly to the rehabilitation, for 
example indicating correct (or incorrect) cognitive activation or muscle activation attempts. US 
provisional applications 60/604,615 and 60/566,078, tire disclosures of which arc incorporated 
5 herein by reference, describe examples of using BMG and EEG signals as part of rehabilitation. 

While the present application focuses on audio presentation, in some embodiments of 
the invention, the audio presentation is supplemented by visual presentation. In one example, 
system 100 controls a DVD player or uses a TV set or a set-top box to display instructions 
and/or feedback. Optionally, visual rhythmic signals, for example, color waves are used instead 
10 of audio rhythmic signals, for example for deaf patients. The use of dual mode presentation 
may be helpful in some pathologies. It should be noted that control device 240 itself can serve 
as a feedback device, for example by vibrating ot applying force, thus possibly providing three 
modes of feedback. 

Fig. 3 is a schematic illustration of a supportive input device 300 according to an 
15 exemplary embodiment of the invention. In the embodiment shown in Fig. 3 a user's limb is 
strapped to plates 320 with straps 330. Optionally, the plates are mounted at the end of an 
articulated robot arm 310. In an exemplary embodiment of the invention, computer 170 is able 
to train a user's limb by controlling the movement of the user's limb with articulated robot aim 
310 and/or moving plates 320. 
20 Tvnes of Motion 

In some embodiments of the invention, rehabilitation system 100 is used in training 
users with different ability levels. As illustrated in Fig's 1-3 various devices can be used to 
affect different levels of control in training user 290 according to the ability of user 290. In 
some cases, user 290 has full control over the movement of the limb being trained. In other 
25 cases, user 290 has limited control or is only able to feel movement. In yet other cases, user 290 
is unable to control or feel the limb. 

In an exemplary embodiment of the invention, the ability of user 290 is taken into 
account in defining the type of motion user 290 is required to perform. In some cases, user 290 
is required to repeat a specific motion or perform a specific motion, for example a circle or 
30 straight line. When the motion is being generated (by the user or the system) a musical or 
auditory cue and feedback is optionally given. The user rhythm is optionally measured and can 
be corrected or the user can be asked to correot it. Optionally, rehabilitation system 100 is 
programmed to be passive and allow any user motion. Alternatively or additionally, 
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rehabilitation system 100 may offer resistance at various levels to the user's motion. In some 
embodiments of the invention the resistance maybe uniform over a trajectory performed by the 
user. Alternatively or additionally, the resistance may act like a force field offering varying 
resistance at different points of the force field In other cases rehabilitation system 100 is 
5 programmed to assist the user in moving his or her limb along a selected trajectory, for 
example once an initial motion in a correct direction was applied. In some cases system 100 
corrects user mistakes, for example to within a predefined tolerance (e.g., if motion is in a 
generally correction direction, it is constrained to the exactly correct direction). In some cases 
rehabilitation system 100 may force the user to perform a specific motion. In one example, both 

10 the tempo of the music and the force applied by system 100 are directed towards causing a user 
to perform a motion at a desired velocity profile. In other cases, system 100 may apply force 
only if a user strays from a trajectory more than a given amount. In other cases, system 100 may 
remind the user of a required motion by providing a nudge. In some cases, system 100 serves to 
smooth user motion. Optionally, the intended motion is detected by measuring force applied by 

15 the user against control device 240. 

In an exemplary embodiment of the invention, if input device 160 is assisting, resisting 
or forcing a user to perform motion rehabilitation system 100 measures the level of resistance 
of the user to the required motion, for example the measure of force the user exerts in different 
directions. 

20 In an exemplary embodiment of the invention, the degree of resistance/assistance may 

be changed, for example, to determine a user's ability. Optionally, music used as a tempo or 
rhythm for rehabilitation is changed, for such testing. 

Optionally, the quality of motion and/or a patient's ability are measured on a musical 
scale, for example, of accuracy, clarity, and/or pleasantness. 

25 Instruction 

In an exemplary embodiment of the invention, a user is instructed visually, for example 
by the required motion being displayed on a display device, for example display 120 or a TV 
screen. Alternatively or additionally, the required motion is requested veibally, for example 
"move left hand up and down". In some embodiments of the invention the user can perform 
30 any type of motion as long as it correlates to attributes dictated by the music, for example 
tempo or path interval. Optionally the chosen user motion may be limited by input device 160, 
for example the user may move in any direction as long as he or she is grasping input device 
160, which may apply a constraint on motion, for example a position and/or velocity constraint. 
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In an exemplary embodiment of the invention, the user motion is displayed to the user 
as movement in a virtual world, for example a user's limb is shown as a controllable object on 
a display (e.g., goggles or a helmet) showing the virtual world. Optionally the virtual world 
may provide tasks which need to be performed by the user, for example touching certain 
5 objects or pushing objects with a minimum force. 

A particular feature of some embodiments of the invention is the ability to bypass 
damaged neuronal pathways* In an exemplary embodiment of the invention, a user is instructed 
using music, rather than, or in addition to verbally and/or visually. In an exemplary 
embodiment of the invention, a stroke victim is taught a relationship between music and 
10 motion, for example by allowing motion and playing different music to match each motion. A 
more structured approach may be used as well, &t example, moving a limb and playing 
suitable music at the same time. In an exemplary embodiment of the invention, system 100 
coordinates the movement with the music, for example playing a predetermined music for each 
motion detected by it. Optionally, the effect of speed, smoothness, tremor and voice are also 
15 taught as affecting musical playback, for example, affecting one or more of tempo, one or more 
channels, amplitude, type of instrument and/or special effects (e.g., echo). 

Once taught the rudiments of a musical language, physical rehabilitation can proceed in 
parallel with any speech rehabilitation. 

In an exemplary embodiment of the invention, system 100 is designed to operate under 
20 voice control. In one example, the system is strained to recognize simple commands, like 
"stop", "slow down", "next" and "repeat". Optionally, the system recognizes musical 
commands, for example, whistling or rhythmic motion of control device 240. Alternatively or 
additionally, spatial gestures using device 240 ate recognized. Optionally, musical feedback is 
provided to assist a user in determining that he made a correct gesture, while the gesture is 
25 being made and without necessarily requiring looking at the moving limb. 

In an exemplary embodiment of the invention, instruction is provided by control device 
240 moving, possibly while the user holds on to it. As will be described below, such a 
demonstration may also be accompanied by a musical playback. 
Using Music for Feedback 
30 Fig. 4 is a flow diagram 400 of a method of rehabilitating a user using system 100, 

according to an exemplary embodiment of the invention. In an exemplary embodiment of the 
invention, a user is positioned (410) relative to rehabilitation system 100. in some cases, this 
may require strapping in. In some embodiments of the invention, the user optionally logs on 

9 
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(420) to computer 170 of rehabilitation system 100, so that system 100 can keep track of a 
user's progress, for example if the user is perfonning the required motion more accurately as a 
result of continuous practice. Alternatively rehabilitation system 100 is intended for a single 
user or a user supplies his or her personal data, for example on a diskette or other types of 
5 removable media. Optionally a user logs in using a smart card. Data regarding the user may be 
stored at a remote location. Optionally, the rehabilitation session is managed over a network, 
for example a local network or a wide area network. 

At 430, the user selects a rehabilitation program that is required for the user, for 
example the user may select between different stress levels or programs for dealing with 
10 different limbs. Optionally, the user selects between programs that perform different motion 
trajectories, for example moving in a curved path (e.g., a circle, an ellipse) or moving in a 
straight path (e.g., a line, a rectangle or forward and back). As can be appreciated, in some 
cases, some of the above tasks of positioning, logging in and selecting an exercise may be 
performed by an operator, rather than by a patient The selection of available exercises may be 
15 set in many ways, for example, automatically, according to a progress plan of the patient or 
manually, by an operator. Various rehabilitation parameters may be set, for example, limits, 
accuracy required, repetitions and/or any parameter known in the art. 

hi some embodiments of the invention, the rehabilitation program may be represented 
by a specific music, which the user is required to create by performing a motion matching the 
20 tempo of the selected music, for example moving at a specific rate and/or following a specific 
path. In some embodiments of the invention, the required motion is displayed to the user on 
display 120 (or control device 240 demonstrates it by moving itself) and the user is required to 
imitate the displayed motion. Music is optionally played during the demonstration and the same 
music is optionally generated if (and as) the user repeats the demonstrated motion. 
25 At 432, one or more music parameters are optionally selected, for example, including 

one or more of music type (for example, quiet music or loud music), linkage of musical 
parameters to rehabilitation parameters (e.g., physiological or motion parameters). 

In an exemplary embodiment of the invention, the music or class of music defines the 
stress level by defining a faster or slower tempo and/or a larger or smaller interval of motion, 
30 which is optionally related to attributes of the music. 

In addition, music for setting an atmosphere maybe selected. 

Optionally, rehabilitation system 100 plays (440) the selected music. The user is 
required to perform (450) a motion which is tracked by rehabilitation system 100 according to 

10 
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methods mentioned above. Optionally, the required motion and the music are selected and/or 
modified to match (and optionally provide guidance), for example as will be described below. 
In other embodiments, music is played only in response to a user motion, rather than during 
motion and/or being modified by the motion. In some cases, the user is not required to carry out 
5 a particular motion, rather any motion he carries out may have constraints set thereon, or free 
motion maybe allowed. 

In an exemplary embodiment of the invention, the music is synchronized to motion of 
control device 240, in that correct motion will create correct music. In another example, the 
motion of device 240 is made to match that of the music. In another example, device 240 

1 0 assists, nudges and/or corrects a user motion so that it will match the music. 

In an exemplary embodiment of the invention, rehabilitation system 100 measures and 
optionally records (460) the user motion as described above, for example by physical contact 
with an input to rehabilitation system 100 (e.g., a user grasping input device 160) or by non- 
physical contact with an input to rehabilitation system 100 (e.g., a camera recording the motion 

15 from a distance). 

hi an exemplary embodiment of the invention, rehabilitation system 100 analyses (470) 
the recorded motion relative to the required motion to determine a level of correlation. In some 
embodiments of the invention, the analyzed results aTe logged for each user to deiermine their 
relative ability and/or if the user shows improvement. 

20 In an exemplary embodiment of the invention, rehabilitation system 100 issues (480) a 

response to the user motion. In some embodiments of the invention, deviation from the 
required path is followed by an immediate response, for example distortion of the music being 
played and/or additional audio sounds such as beeps or other noises. Alternatively or 
additionally, rehabilitation system 100 may respond with a report after finishing an exercise 

25 session regarding the user's ability. In some embodiments of the invention, rehabilitation 
system 100 issues a* report based on details from multiple uses of the device by a user. The 
report optionally deals with the improvement or lack of improvement of a user. 

Immediate responses can be, for example discrete or continuous. In an example of a 
discrete response, a tone is played when a motion segment is correct or incorrect. In an example 

30 of a continuous response, a tempo relates to the speed of motion. In another example of a 
continuous response, the motion is construed to reflect a portion of music and system 100 plays 
a matching counter point Possibly, two sets of feedback are thus available, one set in the music 
corresponding to the motion and one in the counterpoint music. 

11 
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In an exemplary embodiment of the invention, a wave (or other format) file is pre- 
selected for each motion and/or for each position in space. If the position/motion is not reached 
exactly and/or on time, a distorted file will be played. Optionally, the distorted file includes a 
hint for one or more of amplitude of the error and direction of error. 
5 It should be noted that not all parameters of the motion need to be responded to using 

music. For example, only one ot more of the possible parameters are related to music. In one 
example, tempo is used for feedback on ike speed of the motion, but the position is analyzed 
and presented using visual or kinesthetic means. 

Tn an exemplary embodiment of the invention, the method of Fig. 4 is used for special 
1 0 rehabilitation exercises, for example aim rotations. Alternatively or additionally, this method is 
used for rehabilitation of daily activities. For example, different pieces of music may be 
associated with different activities, such as brushing teeth, tying shoelaces or walking. The 
music may serve as a reminder to the patient after rehabilitation is completed. 
Using music to gnide 

15 Fig. 5 is a flow diagram 500 of an alternative method of rehabilitating a user with 

rehabilitation system 100 in accordance with an exemplary embodiment of the invention. As in 
Fig. 4, a user is positioned (510) relative to rehabilitation system 100 and optionally logs on 
(520). After logging on, the user optionally selects (530) a specific rehabilitation program to 
perform. 

20 At 532, musical parameters are selected to match the program. In an exemplary 

embodiment of the invention, the parameters are selected in a way which will guide a user, for 
example as described below. Selection may be, for example, automatic, from a table or 
manually. 

At 534, the music is played and serves as a guide for a user performing a motion (540). 
25 In an exemplary embodiment of the invention, the music serves as a guide in one or more of the 
following manners: 

(a) Tempo is used to tell user, for example, how fast to move, relative motion speed? 
(self calibration) and/or when to move. Tn particular, background music/tempo can be used to 
set the pace for the whole exercise or change during exercise as the user gets wanned up and/or 
30 slow down when end is approaching. The user being warmed up may be detected, for example 
using physiological parameters, or based on time considerations. Also background music can 
be used to add interest during rest periods and/or during an exercise. Optionally, the music 
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slows down when the patient stops and speeds up when motivation to try again is to be applied 
(e.g., decided manually or automatically). 

(b) Feedback in general, for example as described with respect to Fig. 4, in that music is 
distorted by incorrect movement. 
5 (c) Guidance on expected length of exercise and on time to do repetitions (for repetitive 

music). 

(d) Training, in which music is used to train in doing exercises. For example, different 
musical phrases can be different motions which patient learns. In another example, a set of 
morning exercises (tai chi) are taught using matching music. 
10 (e) Synchronizing different body parts, for example, by a particular phase indicating 

combined motion or by first teaching each separate motion with its own music and then 
combining the motions and the music. 

(f) Anticipation, for example by generating a tone or changing amplitude before a 
change of motion is required. 
15 (g) Instruction, for example, particular musical phrases meaning "go to starting point", 

"faster" or "time to slow down". 

(h) Warning. Music can be used to warn a user that he is approaching a point where he 
earlier felt pain or a point where a joint is unstable and care should be taken. 

hi some embodiments of the invention, the user's motion is measured (550) by 
20 rehabilitation system 100. Optionally, rehabilitation system 100 analyzes (560) the motion to 
match it to music and/or sounds based on the user's motion, for example the position of the 
user's limb, the rate of motion and/or acceleration. Exemplary feedback methods were 
described above. Optionally, in method 500, music and/or additional sounds are created (570) 
and optionally played for the user on the fly according to the user motion, for example music of 
25 a selected song is recreated with a tempo based on the rate of the user's motion. Tn another 
example, the movements emulate playing on a musical instrument. 

In an exemplary embodiment of the invention, guidance and/or feedback are used to 
give a patient a sense of time and/or a sense of timing. 
Record and playback 

30 In an exemplary embodiment of the invention, motions are programmed into system 

100 by an instructor, in combination with music. For example, an instructor can show a correct 
motion and match it to music after the fact or perform to music. In an exemplary embodiment 
of the invention, position sensors (e.g., as shown in Fig. 2) are attached to the instructor and the 
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position of various body parts recorded. In another example, the instructor moves the patient's 
limbs in a demonstration which is recorded, for later repetition by the patient In an exemplary 
embodiment of the invention, the position sensors include electrical or vibrational stimulators, 
hi one example, the location change of each position sensor is recorded to a music-look 
5 up table of the following (or similar) form: 



Tune (name) 


File name 


Tempo 






Yesterday 


Yestwav 


XXX 






Joint identifiers 


Joint 1 


Joint 2 


Joint 3 


Joint x 




Left elbow 


Right elbow 


Left knee 


Right knee 


Musical Bar# 


Joint 1 


Joint 2 


Joints 


Joint x 


1 


No motion 


+ 






2 




No 






3 




no 







Where V indicates motion away form the body. 



During playback, the patient plays the music (e.g., thru earphones). Controller 170 
signals to each stimulator based on the above table and the patient responds accordingly, 

10 Position sensing and/or accelerometers may be used to determine if the motion is correct and 
optionally provide musical feedback 
Integrated motion controller 

In an exemplary embodiment of the invention, a musical rehabilitation/exercise device 
is made portable. Such a device includes a controller and a plurality of stimulation patches. 

15 Headphones, optionally wireless are used to play music. The controller stimulates the patches 
according to the desired response by the user. Optionally, patches are provided in pairs or in 
greater numbers, for example for providing stimulation to a joint from multiple directions. 
Optionally, the patches include position sensors and/or accelerometers (or such are separately 
provided) for porting to the controller the patient's response. In an exemplary embodiment of 

20 the invention, such a device is used for assistance in daily activities, with the stimulation and 
the music serving as reminders for the motions to be carried out. The stimulators and/or sensors 
may be provided in an article of clothing, for example a vest and/or be wireless. Optionally, the 
stimulation is electrical or vibrational. 
Body synchronization 

25 As noted above, in an exemplary embodiment of the invention, musical means are used 

to help a patient synchronize motions between different body parts. In one example, a 
rehabilitation process is broken down into separate processes for each body part or for different 
motions of a same body part. Then a complex motion is guided by playing the music of 
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multiple motions serially or in parallel, as appropriate. Delay in one channel may be used to 
indicate that its corresponding motion should be speeded up. In an exemplary embodiment of 
the invention, different body parts and/or motion are provided with different instrumental 
sounds. 

5 In an alternative embodiment, music is used to indicate specifically the synchronization. 

For example, a piece of music is played correctly only if synchronization is correct, possibly 
using the same instruments for each body part as using during training. As noted above, the use 
of certain musical themes may be standardized so as to create a musical language, fa one 
example, left leg is always the sound of a flute and a right leg always an organ. 

10 In an exemplary embodiment of the invention, rehabilitation system 100 and/or input 

device 160 may record motion from two limbs (e.g. arms or legs) simultaneously. In an 
exemplary embodiment of the invention, the user is required to perform an identical motion 
with both limbs. Alternatively or additionally, the user is required to perform mirrored motion, 
where the two limbs axe required to perform miirored motion. Optionally, the user is required 

15 to perform complementaiy motion or motion with a correlation between different limbs, for 
example throw an object from one hand to another. 
Instrument alavfng 

In an exemplary embodiment of the invention, music i$ generated as if a user had one or 
more instruments. This may be part of a free-play session or it maybe part of an exercise where 

20 particular motions are mandated. In an exemplary embodiment of the invention, the created 
music are selected according to the type of motion, for example a user can imitate playing a 
specific instrument such as a drum, guitar, violin, flute or any other instrument with a distinct 
playing pattern, In an exemplary embodiment of the invention, the various instruments are 
"positioned" in space, either as real instruments or as locations where they can be played by 

25 suitable motions. In an exemplary embodiment of the invention, a user plays such instruments 
in sequence, for example as set up by a therapist. Optionally, background music is provided to 
which such playing is to be synchronized. Optionally, the instruments are visible using virtual 
reality methods. Alternatively or additionally, the instruments include a light or other visual 
indication to remind a user when to "play" them. Optionally, an audible indication is used. 

30 Optionally, this allows different spatial locations to be mapped to musical sounds and/or 
instruments, as part of a rehabilitation language. In a session with pre-set motions, the musical 
generation may be made relative to the motions (e.g., to one or more of spatial, acceleration, 
force and/or velocity parameters. Alternatively or additionally, the generation is fixed in space. 

15 
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Optionally, as a user improves, the complexity of music created is increased. Alternatively or 
additionally, me degree of assistance is reduced. For example, a starting user may have a whole 
phrase generated just for making a motion, as time progresses, the user may get to a point 
where only a single note is generated and only if a correct motion is made). Exemplary 
5 intermediate levels are playing of a set of instruments, playing of a cord and triggering 
playback of channels. 

In some embodiments of the invention, other parameters of me created music are 
selected by other aspects of me user motion, for example the height at which the motion is 
performed. 

10 In an exemplaiy embodiment of the invention, control device 240 is used as a musical 

instrument. In one example, different locations, motions, speeds, and/or other musical 
parameters can make musical sounds. la an exemplary embodiment of the invention, the plane 
is mapped to various tones and the Z position is mapped to octave. Similarly, a mapping may 
be made in spherical coordinates. Optionally, the mapping matches a physiological motion. For 
15 example, the plane to which tones are matched is the surface of the edge of motion of a hand, 
with the octave being set based on the degree of bending of the elbow. This can give a user a 
feeling for range of motion and change in range of motion, especially if there is a problem with 
self-kinestheac feedback. 

In a tempo-teaching example, a user can reach for an instrument and a tone is generated 
20 • only if the movement matched the tempo. Optionally, a time constraint is applied. Optionally, 
rehabilitation using music can be used to single out individual issues to be rehabilitated. For 
example, a user can perform an exercise whose purpose is to make a movement (e.g., any 
movement, a semi-correct movement, an exactly correct movement) at a given time. 
Conversely, rehabilitation can focus on the motion and provide music relating to the 
25 correctness of motion (e.g., music is fine as long as motion is within bounds) without training 

timing at the same time. 

In a more complex example, a drum machine may be emulated, where different drums 
and other percussion instruments are positioned in different target volume of space. The 
applied force is easy to portray using amplitude. 
30 Two drum sticks or a stick and a pedal (e.g., with position sensors) may be used for 

training two body parts simultaneously. System 100 can be used to demonstrate or instruct the 
user online on how to perform the music correctly. In general it should be noted that system 
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100 can stop a session and restart it or provide additional instruction or motivation, if results 
achieved by a user indicate it would be useful 

Multi-user app lications 

In some embodiments of the invention, rehabilitation system 100 coordinates between 
5 multiple users, for example allowing competition between them or cooperative game playing. 
Optionally, the motion of different users can be assisted or hindered in order to reach a 
common level of complexity for user's with varying ability. In some embodiments of the 
invention, a single rehabilitation system 100 can track multiple limbs and/or users, for example 
using a single camera and/or a single computer 170. Alternatively or additionally, multiple 
1 0 input devices 160 are used hi some embodiments of the invention, multiple computers 170 are 
used Optionally, multiple users can practice or compete together over a communication line, 
for example multiple users each with a rehabilitation system 100 can participate in a common 
virtual reality environment over a LAN or die Internet. 

A particular example of cooperation is in music making. Each user can have a different 
1 5 instrument. Optionally system 1 00 provides support (for example correction of movements) so 
that even severely impaired patients can take part. A potential advantage of such a setup is that 
impaired people can feel useful and take part in socisd activities, where a realistic level of 
participation is expected of them. Such a level is optionally determined by a therapist or 
automatically based on performance in exercises. 
20 In an exemplary embodiment of the invention, music is used in a group therapy so that 

it is easy to portray to other users what a user did wrong. This is especially true if there are 
physical problems, such as the users being distributed and having a limited bandwidth and/or 
no cameras. Alternatively or additionally, music is used to help explain what motions were 
done. 

25 hi another example, a therapist or a computer acts as a conductor to guide a group 

session. In one example, a control device 240 is moved and music is sounded and the users 
repeat This can he done in a band setting or not. As the users show their ability, the process 
may be made more complex and/or speeded up. As noted above, by providing support to weak 
users, such weak users can partake also if complexity increases by system 100 hiding such 

30 complexity and/or increasing a correction function of system 100. 

Tn some cases, the users compete with each other, for example, to see who can create 
clearest music or while playing a game. Optionally, system 100 is used to even die odds 
between users of different abilities. 
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Healthy-and-patient example 

A particular example where music can be used to synchronize between multiple users is 
a therapist and a patient, where the music is used to synchronize their motions and/or allow the 
therapist to focus on a smaller number of mistakes in the motion, with other mistakes being 
5 pointed out" by the music. 

Another example is a home setting, where, for example, a layperson assists an elderly 
parent in rehabilitation. System 100 can compensate for lack of knowledge by providing music 
which guides both the healthy person and the parent-patient The music serves to indicate what 
is correct and what should be corrected and the layperson can focus on caring and on helping 
10 such corrections to be carried out. It is believed that system 100 can thus reduce frustration and 
increase closeness in families with crippled members. 
Musical variations 

As used herein, music is any type of audio with rhythm and can include, for example, 
chanting, and reading of prose, as well as periodic beeping. It is believed, however, the 
15 complex music and interesting music can have special benefit in providing multiple feedback 
possibilities, detailed feedback and/or interest Inparticular, music can range between children's 
rhymes and complex orchestral pieces, for example, rock music, folksongs and pieces 
composed for a small number of instruments (e.g., 1, 2, 4 or 8). 

In an exemplary embodiment of the invention, music is selected according to user 
20 preference. For example, a plurality of groups of music are available and a user selects one 
group. For example, groups can be "Rock", "Pop", Baroque" and "Chanting". Each group 
includes elements which can be used for the various uses described herein. 

In an exemplary embodiment of the invention, music is selected for a particular 
rehabilitation task based on one or more of pace, beat, number of channels, regularity, 
25 existence, number and/or location of clearly identifiable turning points and liking and/or 
familiarity of the patient. 

As noted above, different users have different abilities, so the actual parameters 
controlled by the user in a particular session may vary. For example, the parameters used may 
depend on one or more of: complexity of exercise, proficiency of user in music, rehabilitation 
30 stage of the user, ability of therapist, presence of participating group and/or mood. 

In an exemplary embodiment of the invention, it is appreciated that a user is in a 
learning process when using music. Thus the complexity and/or speed of the music may 
increase over time. In an exemplary embodiment of the invention, the complexity, duration, 
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type and/or difficulty of rehabiUtation exercises increase over time, possibly at a different rate 
than music. In an exemplary embodiment of the invention, the number of instruments, runes 
chords and/or cords used during rehabilitation is gradually increased. Optionally, additional 
musical variation (e.g., additional channels or instruments in an already existing musical piece) 
5 is introduced when a user shows advancing rehabilitation proficiency in one example, a vocal 
channel is added to provide feedback on smoothness of motion, once a user has shown 
proficiency in an exercise in which an instrument channel is linked to accuracy in controlling 
direction of motion. 

hi an exemplary embodiment of the invention, fixed parameters of music are mapped to 
10 rehabilitation parameters, for example, smoothness is linked to random variation in amplitude 
and accuracy of motion is linked to accuracy in reproduction of amplitude, frequency and/or 
tempo. As noted, different exercise may have different mappings. For example, in one exercise, 
any error smaller than 3 cm in position is ignored. In another, the effect of error on distortion of 
music is linear. In another, the effect is non-linear, for example exponential. The effect may be 
15 directional (e.g., increase in one direction and decrease in the other) or urn-directional, for 
example decrease in any direction away from correct direction. The term directions is used to 
represent any motion parameter, including one or more of position, velocity acceleration and 
force. Different effects maybe used for patients with different disorders and/or depending on 
their stage or type of rehabilitation exercise used. 
20 hi some cases, music is used for psychological effect, in which case 1he effect of 

particular music on the patient is optionally gauged in a calibration session or over time. In one 
example, music is used to calm a patient who is over exerting. In some cases the effect is direct 
on the patient's emotions, in others the music serves as a signal to the patient (or therapist), or 
as a mirror of his emotional response. 
25 In an exemplary embodiment of the invention, a user can provide his own music, for 

example a favorite music provided on CD. Such music can be used as background music or for 
a Karoke game. In an exemplary embodiment of me invention, the music is analyzed (e.g., 
divided into channels) for use in rehabilitation. Optionally, the variations in the music which 
signify feedback and/or guidance are illustrated to the user prior to or during rehabilitation. 
30 Games 

In an exemplary embodiment of the invention, music is used to provide fun during 
rehabilitation. In some embodiments, system 100 is used to support games, for example, single 
player games or multi-player games. 
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One example of a game is "Hit on Beat" in which a player gains points (or plays a 
sound) by reaching a desired spatial location and/or speed and/or acceleration and/or force at a 
time that coincides with a beat of music and/or before a musical timer runs out. Repeated "hits" 
maybe supported. 

5 Another example is "Musical Simon" . Control device 240 makes a motion while patient 

290 holds on and also sounds music at a same time. In a later level, possibly only the sound 
will be provided and the user will need to reconstruct the motion from memory. Only repeating 
correct motion (e.g., particular parameters such as position, velocity or others) and a correct 
sequence, will provide points. The motion may be constrained and/or assisted as described 
10 above. In a variant, the game is discrete and particular locations in space are associated with 
music. Repeating a set of musical tones requires a correct spatial motion sequence. Graphics 
are optionally used for visualization. In one example, a patient uses VR (or TV screen with 
graphical representation of motion), with different musical instruments each corresponding to a 
spatial target. The target size and shape and position can depend on the desired rehabilitation. 
15 in a particular example, system 100 is used as a courser input device, with cursor 

positions mapped according to a particular rehabilitation exercise (e.g„ drawing a circle, 
flexing an arm, etc.). The rules can be as follows: The game displays four different graphical 
signs. The player needs to replay the tracks by the moving the cutsot to the graphical signs in 
the same order. Each iteration the series of tracks grows one track longer, making it harder to 
20 replay the tracks in the correct order. For each good repeat the player will be awarded 1000 
points. For out-of-tempo Tepeat he will be awarded 500 points. After the first wrong repeat the 
game will end and display a Game Over message and the score. The goal is to win as many 
points as possible. This is done by remembering as many musical notes as possible. The 
difficulty level can determine the music tempo. 
25 In an exemplary embodiment of the invention, an interface module (software and/or 

hardware) is provided so that many cursor or joystick controlled games can be controlled using 
input device 160. The game maybe on a same or on a different computer. 

In general (for games and otherwise), exercises can be made more difficult by 
elongating the chain of causality between the instruction to the user and his performance. A 
30 shortest chain is the patient repeating a motion shown by a control 240. Intermediate is trying to 
reproduce a sound sequence (though sounds can be used as a hint to assist in a short-chain 
example). A still longer chain is a patient being taught spatial locations corresponding to 
sounds or instruments and expecting the patient to chain them together (e.g., plan motion 
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trajectories) in response to hearing and anticipating a piece of music. This can train the speed 
and other characteristics of motion trajectory planning. In one example, some musical channels 
are provided and the patient provides the others. In another example, only words are shown and 
the patient needs to provide the music or only parts of it, for example a rhythm (absolute or 
5 relative values). As can he appreciated, music and motion control can be used to assist in 
cognitive rehabilitation, for example, for shape (^crimination (e.g., recognizing shapes based 
on hand being moved along their outline and being shown), verbal processing (matching words 
to rhythm and/or motion), planning (e.g., of motion) and symbol sequencing (by doing ordered 
motion). 

10 Another example of a game is Karaoke - a background tube is played while the user 

plays to the tune by one or more of: moving to a target instrument in space; moving between 
target instruments in space (a set of instruments is optionally placed by the therapist (or user or 
computer in space)); by just moving in the right direction; and/or by using multiple limb for 
more than one (VIRTUAL) instrument 

15 hi an exemplary embodiment of the invention, games and/or music arc used for 

rehabilitation of children- However, simpler music may need to be used for younger children. 
Additionally, music and common childhood games may be played with the assistance of 
rehabilitation system 100. Optionally, children group games are provided. Optionally, children 
are outfitted with sensors so that their motions can be tracked and responsive music generated, 

20 In an exemplary embodiment of the invention, music feedback and/or guidance is used 

to rehabilitate mentally damaged children, for example, children with Down's syndrome or 
children with Cerebral Palsy. 
Portability 

In some embodiments of the invention, rehabilitation system 100 is constructed from 
25 elements which are lightweight and/or compact in order to allow portability of the device. 
Optionally, the elements of rehabilitation system 100 are easily assembled and calibrated so 
that a user is not restricted to a specific location. In some embodiments of the invention, 
rehabilitation system 100 is supplied as a kit with all necessary parts, for example folding 
tables and/or poles to position cameras, in order to limit dependency of rehabilitation system 
30 100 on external elements of a specific location. 
Feedback 
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In an exemplary embodiment of the invention, the user receives immediate feedback, 
for example a distortion in the music being played or created responsive to the motion being 
performed. Optionally, delayed feedback is provided as well. 

In some embodiments of the invention, rehabilitation system 100 supplies a user with a 

5 report regarding the user's performance relative to the requirements even for a single session. 
Optionally, rehabilitation system 100 supplies long term performance reports regarding a user's 
progression over a period of many sessions. In some embodiments of the invention, 
rehabilitation system 100 supplies reports regarding a user's performance relative to other users 
and/or an average user. In an exemplary embodiment of the invention, progress Teport are 

10 provided as musical files. In this way, a cognitively impaired and/or layman patient can be 

easily shown his progress. 

In some embodiments of the invention, feedback reports can be accessed remotely by a 
user's therapist in order to monitor a user's progress, for example. Optionally, rehabilitation 
system 100 may send reports to a common database or directly to a therapist, for example by 
15 Email. 



The use of music for rehabilitation is not limited by the particular examples shown 
above. In particular, music can be used, for example for the fine motor control rehabilitation 
methods described in US provisional application 60/566,079, the disclosure of which is 
20 incorporated herein by reference. For fine motor control, music can be used to provide 
feedback for a task which the user is not supposed to be looking at, such as writing or eating 
and in which music is used to indicate if a motion is correctly executed. 

Music can be used for neural rehabilitation. For example, US provisional application 
60/604,615, the disclosure of which is incorporated herein by reference uses neuronal sensing 
25 to determine when an action should be triggered. EEG signals can be used to modify music so a 
patient can recognize his neural activity. 

Music can be used with EMG. For example, US provisional application 60/566,078, the 
disclosure of which is incorporated herein by reference musical signals can be used in addition 
to EMG or to prompt the patient that EMG is about to be or is being delivered. This may 
30 increase sensitivity. 

Music can be used with gait training, for example such as described in U.S. Provisional 
Application filed on even date with the instant application, titled "Gait Rehabilitation Methods 
and Apparatuses", attorney docket number 414/04173, filed on even date and with a common 
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inventor, the disclosure of which is incorporated herein by reference. In one example, music is 
used to set the pace of walking or to synchronize the multiple body motions. In another 
example, music is used to help a patient tell which one of the body motions (e.g., hips, legs, 
feet, shoulders) and on which body side (e.g., using stereo) is out of synch. 
5 Music can be used for balance training, for example such as described in U.S. 

Provisional Application filed on even date with the instant application, titled "Methods and 
Apparatus For Rehabilitation And Training", attorney docket number 414/04212, filed on even 
date and with a common inventor, the disclosure of which is incorporated herein by reference. 
In one example, music is used to indicate balance between body sides. A channel which is too 
10 loud may be used to indicate a body portion applying too much force. Silence may be used to 
indicate balance, while a wobble will generate a cyclical tube and as balance is lost and alarm 
may increase in amplitude. 

It should be appreciated that the above described methods of rehabilitation may be 
varied in many ways, including, omitting or adding steps, changing the order of steps and the 
15 types of devices used. In addition, a multiplicity of various features, both of method and of 
devices have been described. In some embodiments of the invention, mainly methods are 
described; however, also apparatus adapted for perfonning the methods are considered to be 
within the scope of the invention. It should be appreciated that different features may be 
combined in different ways. In particular, not all the features shown above in a particular 
20 embodiment are necessary in every embodiment of the invention. Further, combinations of the 
above features are also considered to be within the scope of some embodiments of the 
invention. Also within the scope of the invention are kits which include sets of a device, one or 
more attachments and/or software. Also, within the scope is hardware, software and computer 
Teadable-media including such software which is used for carrying out and/or guiding the steps 
25 described herein, such as control of arm position and providing feedback. Section headings are 
provided for assistance in navigation and should not be considered as necessarily limiting the 
contents of the section. When used in the following claims, the terms "comprises", "includes", 
"have" and their conjugates mean "including but not limited to". It should also be noted that the 
device is suitable for both males and female, with male pronouns being used for convenience, 
30 ft will be appreciated by a person skilled in the art that the present invention is not 

limited by what has thus far been described Rather, the scope of the present invention is 
limited only by the following claims. 
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CLAIMS 

1 . Apparatus, comprising: 

at least one sensor which generates an indication of a motion of a user; 
5 a rhythmic audio generator; and 

a controller configured to control said generator to modify an audio output thereof 
according to a program of rehabilitation of a human, 

2. Apparatus according to claim 1, wherein said controller modifies said audio according 
10 to said indication. 
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